Hyperparameter Optimization of Deep Neural Networks: Combining Hyperband with Bayesian Model Selection
نویسندگان
چکیده
One common problem in building deep learning architectures is the choice of the hyper-parameters. Among the various existing strategies, we propose to combine two complementary ones. On the one hand, the Hyperband method formalizes hyper-parameter optimization as a resource allocation problem, where the resource is the time to be distributed between many configurations to test. On the other hand, Bayesian optimization tries to model the hyper-parameter space as efficiently as possible to select the next model to train. Our approach is to model the space with a Gaussian process and sample the next group of models to evaluate with Hyperband. Preliminary results show a slight improvement over each method individually, suggesting the need and interest for further experiments.
منابع مشابه
Combining Hyperband and Bayesian Optimization
Proper hyperparameter optimization is computationally very costly for expensive machine learning methods, such as deep neural networks; the same holds true for neural architecture search. Recently, the bandit-based strategy Hyperband has shown superior performance to vanilla Bayesian optimization methods that are limited to the traditional problem formulation of expensive blackbox optimization....
متن کاملCombination of Hyperband and Bayesian Optimization for Hyperparameter Optimization in Deep Learning
Deep learning has achieved impressive results on many problems. However, it requires high degree of expertise or a lot of experience to tune well the hyperparameters, and such manual tuning process is likely to be biased. Moreover, it is not practical to try out as many different hyperparameter configurations in deep learning as in other machine learning scenarios, because evaluating each singl...
متن کاملPractical Hyperparameter Optimization
Recently, the bandit-based strategy Hyperband (HB) was shown to yield good hyperparameter settings of deep neural networks faster than vanilla Bayesian optimization (BO). However, for larger budgets, HB is limited by its random search component, and BO works better. We propose to combine the benefits of both approaches to obtain a new practical state-of-the-art hyperparameter optimization metho...
متن کاملHyperband: Bandit-based Configuration Eval- Uation for Hyperparameter Optimization
Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While recent approaches use Bayesian Optimization to adaptively select configurations, we focus on speeding up random search through adaptive resource allocation. We present HYPERBAND, a novel algorithm for hyperparameter optimization that is simple, flexible, and theoretically sound. HYP...
متن کاملFast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
Bayesian optimization has become a successful tool for hyperparameter optimization of machine learning algorithms, such as support vector machines or deep neural networks. Despite its success, for large datasets, training and validating a single configuration often takes hours, days, or even weeks, which limits the achievable performance. To accelerate hyperparameter optimization, we propose a ...
متن کامل